Dynamic-Depth Context Tree Weighting
نویسندگان
چکیده
Reinforcement learning (RL) in partially observable settings is challenging because the agent’s observations are not Markov. Recently proposed methods can learn variable-order Markov models of the underlying process but have steep memory requirements and are sensitive to aliasing between observation histories due to sensor noise. This paper proposes dynamic-depth context tree weighting (D2-CTW), a model-learning method that addresses these limitations. D2-CTW dynamically expands a suffix tree while ensuring that the size of the model, but not its depth, remains bounded. We show that D2-CTW approximately matches the performance of state-of-the-art alternatives at stochastic time-series prediction while using at least an order of magnitude less memory. We also apply D2-CTW to model-based RL, showing that, on tasks that require memory of past observations, D2-CTW can learn without prior knowledge of a good state representation, or even the length of history upon which such a representation should depend.
منابع مشابه
The Context-tree Weighting Method: Extensions - Information Theory, IEEE Transactions on
First we modify the basic (binary) context-tree weighting method such that the past symbols x1 D; x2 D; ; x0 are not needed by the encoder and the decoder. Then we describe how to make the context-tree depth D infinite, which results in optimal redundancy behavior for all tree sources, while the number of records in the context tree is not larger than 2T 1: Here T is the length of the source se...
متن کاملThe Context-Tree Weighting Method : Extensions
First we modify the basic (binary) context-tree weighting method such that the past symbols x1 D; x2 D; ; x0 are not needed by the encoder and the decoder. Then we describe how to make the context-tree depth D infinite, which results in optimal redundancy behavior for all tree sources, while the number of records in the context tree is not larger than 2T 1: Here T is the length of the source se...
متن کاملSkip Context Tree Switching
Context Tree Weighting is a powerful probabilistic sequence prediction technique that efficiently performs Bayesian model averaging over the class of all prediction suffix trees of bounded depth. In this paper we show how to generalize this technique to the class of K-skip prediction suffix trees. Contrary to regular prediction suffix trees,K-skip prediction suffix trees are permitted to ignore...
متن کاملA Context-Tree Branch-Weighting Algorithm
The context-tree weighting algorithm [4] is a universal source coding algorithm for binary tree sources. In [2] the algorithm is modified for byte-oriented tree sources. This paper describes the context-tree branch-weighting algorithm, which can reduce the number of parameters for such sources, without increasing the complexity significantly.
متن کاملA Relationship between Contex Tree Weighting and General Model Weighting Techniques for Tree Sources
This paper explores a relationship between parameters for the context tree weighting and weights for a general model weighting technique. In particular, an algorithm is proposed that approximately computes the parameters from the weights, and a condition under which no error for the approximation occurs is derived. key words: model weighting technique, tree source, context tree weighting.
متن کامل